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(54) Annotation of on-line documents 

(57) An annotation interface is used within a com- 
'ting system. A document (15) is annotated by display- 
ing v. annotation box (51 ) upon a user selecting the an- 
notation feature and a location in the document (15). Af- 
ter a user has added desired text to the annotation box 
(51), ilis Uoer closes the annotation box (51). Tht* se- 
lected location in the document (1 5) is then marked. 



UPON SELECTION IS" ANNOTATE 
FEATURE, INDICATE THE ANNOTATE 
FEATURE 







UPON SELECTING A LOCATION IN THE 
DOCUMENT, DISPLAY ANNOTATION 






UPON CLOSING THE ANNOTATION, 
MARKING THE SELECTED LOCATION IN 
THE DOCUMENT 






UPON SELECTING THE ANNOTATION 
LOCATION. DISPLAYING THE 
ASSOCIATED ANNOTATION 



FIGURE 2 
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Description 
Background 

The present invention concerns annotation of doc- 
uments which are displayed by a computing system. 

When reading a traditional book or a manual, a 
reader may make handwritten notes in the margins. 
These handwritten notes or annotations may include 
questions to be answered later, comments on the text, 
examples of what is discussed, summary information 
and so on. 

For documentation presented on-line, it is generally 
not practical to use handwritten notes to annotate the 
text. Therefore otharwaysmustbefoundtomake notes. 
If maintaining the integrity of the original document is 
not important, a user may use a keyboard to add notes 
directly into the text of an on-line document However, 
for some applications such as on-line books, on-line 
documentation, on-line manuals, etc., it is generally not 
desirable to have the text changed. 

Some attempts have been made to allow the anno- 
tations of an on-line document without changing the 
contents of the document. For example, Stephen R 
Levin, et al sets out an Apparatus for Document Anno- 
tation and Manipulation using Images from a Window 
Source in USPN 5.231,578. In the disclosed system, an 
annotatable document is generated from a window of a 
multi-window support program running independently of 
the annotation program. For each window of the mufti- 
window program, a working document is generated and 
is annotatable by the annotation program. An annotat- 
able document is also generated irwn the whole screen 
view of overlapping windows produced x'ring execution 
of the multi-window support program. Annotated work- 
ing documents may be subsequently stored in and 
thereafter retrieved from the database of the multi-win- 
dow support program. 

Elie Cassoria et al., disclose a Method for Associ- 
ating Annotation with Electronically Published Material 
in USPN 5,146,552. In their system, annotations are 
made and stored separately from the originally pub- 
lished document. The association of annotations with a 
particular context within the document is performed by 
identifying major document elements such as chapters, 
sections, sub-sections, paragraphs, figures, and so on. 
The relative position within the finest" identified docu- 
ment element is then recorded to fix the precise position 
of some annotations that the reader wants to pinpoint to 
a particular line or word position. 

These prior art systems provide some ability to an- 
notate on-line documents. However, the interfaces are 
generally cumbersome. Further, the association be- 
tween annotations and related text is not crisp. It is de- 
sirable, therefore, to develop an annotation system with 
a more elegant interface. 



Summary of the Invention 

In accordance with the preferred embodiment of the 
present invention, an annotation interface for use on a 
s computing system is presented. When using the anno- 
tated interface, a document is annotated by displaying 
an annotation box upon a user selecting the annotation 
feature and a location in the document After a user has 
added desired text to the annotation box, the usercbses 
"> the annotation box. The selected location in the docu- 
ment is then marked. 

For example, in one embodiment of the present in- 
vention, the annotation feature is indicated upon selec- 
tion of the annotation feature. When the annotation fea- 
w ture is indicated, the annotation box is displayed upon 
the user selecting a location in the document The an- 
notation feature is selected by setedhg a software but- 
ton displayedon a display. Alternately, the selection may 
be made, for example, by selecting an "annotation" list- 
20 ing on a pull-down menu, by depressing a particular 
function key on a keyboard, by depressing a particular 
sequence of keys on a keyboard, or by some other 
means. 

The annotation feature is indicated, for example, by 

2S changing a shape of a cursor. For example, the shape 
of the cursor may be changed into the shape of a paper 
clip, a pencil, a scrap of paper, or any other shape which 
is different from the original shape of the cursor. Alter- 
nately, the cursor may be changed by shading the cursor 

3" or placing a drop shadow over the cursor. The annota- 
tion feature may also be indicated in different ways, for 
example, by displaying a predetermined image on tiie 
computer display or making a particular sound by the 
computing system. 

* In the preferred embedment, the user selects the 
iocation in the document,, for example, by placing the 
cursor over the location in the document and making a 
selection using an input device. Also in the preferred 
embodiment, the annotation box is an annotate dialog 

40 box. 

The location is marked, for example, by displaying 
a special icon in a location in a vicinity of the location in 
the document. For example, the location is marked in a 
margin of the document 

45 Upon a user selecting the special icon, the annota- 
tion box is displayed. The text of the annotation may 
then be read, modified and/or deleted by a user. 

The iocaiion of tne special icon may be moved with- 
in the document. For example, a user may move the 

50 special icon by dragging and dropping the special icon 
to a new location. Alternately, the user may move the 
special icon by cutting/copying and pasting the special 
icon to a new location. 

In an alternate embodiment of the present Inven- 

55 tion, the annotation of a document is performed by first 
selecting a location in the document for the annotation. 
A marker is shown in the document at the selected lo- 
cation. Upon the user selecting the annotation feature, 
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the annotation box is displayed. After a user has added 
desired text to the annotation box, the user closes the 
annotation box. The selected location in the document 
is then marked. 

The present invention provides an efficient, intuitive 
and easy to use interface for annotating a document. 

Brief Description of the Drawings 

Figure 1 shows an on-line book displayed on the 
display of a computing system in accordance with a pre- 
ferred embodiment of the present invention. 

Figure 2 is a flowchart which describes operation of 
an interlace for annotating a document in accordance 
with a preferred embodiment of the present invention. 

Figure 3, Figure 4, Figures. Figures, Figure 7, Fig- 
ure 8 and Figure 9 show computer displays which illus- 
trate interaction with an on-line book on a computer 
screen in order to place and manipulate annotations in 
accordance with a preferred embodiment of the present 
invention. 

Figure 10 is a flowchart which describes operation 
of an interface for annotating of text within a document 
in accordance with an alternate preferred embodiment 
of the present invention.. 

Figure 11 and Figure 12 show computer displays 
which illustrate interaction with an on-line book on a 
computer screen in order to place and manipulate an- 
notations in accordance with the alternate preferred em- 
bodiment of the present invention. 

Description of the Preferred Embodiment 

Figure 1 shows a computing system 11 in accord- 
ance with a preferred embodiment of tivs prs3ent inven- 
tion. The computing system includes vatious input de- 
vices such as a keyboard 13 and a mouse 14. Other 
pointing devices may be utilized instead of mouse 14. 
For example, mouse 14 may be replaced or supple- 
mented by a trackball input device, an isopoint input de- 
vice, a trackpad input device, cursor keys or some other 
input device either separate from or incorporated as part 
of keyboard 13. 

Within a computer display 12, a portion of a docu- 
ment 15 is shown. Document 15 contains, for example, 
text and/or graphics. Document 15 is for example, an 
on-line book, on-line documentation, an on-line manual, 
a text file, a graphics file or any document which con- 
tains any combination of text and/or graphics. 

Figure 2 is a flowchart which describes operation of 
an interface for annotating document 15 in accordance 
with a preferred embodiment of the present invention. 
In a step 31 , upon selection by a user of the annotation 
feature, the interface indicates the analog feature has 
been selected. The selection may be performed, for ex- 
ample, by selecting a special "annotation* button. Alter- 
nately, the selection may be made, for example, by se- 
lecting an 'annotation' listing on a pulkJown menu, by 



depressing a particular function key on a keyboard, by 
depressing a particular sequence of keys on a key- 
board, or by some other means. 

The annotate feature may be indicated, for exam- 

5 pie, by displaying a predetermined image on the com- 
puter display or making a particular sound by the com- 
puting system. For example, in the preferred embodi- 
ment of the present invention, the cursor is changed to 
indicate the annotation feature. The cursor may be 

10 changed, for example, by shading the cursor or placing 
a drop shadow from the cursor. Alternately, the shape 
of the cursor may be changed. For example, the cursor 
may be changed into the shape of a paper clip, a pencil, 
a scrap of paper, or any other shape which is different 

»5 from the original shape of the cursor. 

In a step 32, upon a location in the text being se- 
lected, an annotation is displayed. For example, the lo- 
cation is selected by placing the cursor over the location 
and depressing or releasing a key on an input device 

20 such as a mouse, trackball, trackpad or keyboard. The 
annotation is displayed, for example, within an annotate 
dialog box Initially, a displayed annotation does not 
have any user-defined text The user may add the de- 
sired text for the annotation, for example, using a key- 

6 board or another input device. . 

Instead of, or in addition to, adding plain text to the 
annotation box, other information may be attached to 
the analog box. For example, graphics information may 
be attached to the annotation box. The graphic informa- 

30 tion includes, for example, a drawing, map, photo or 
snapshot. In addition to graphics, other types of media 
can be attached to the annotation box. The media in- 
cludes, for example, a sound file, a movie file. 

In a step 33, upon closing the annotation dialog box, 

as the selected location is marked, for example by display- 
ing a special icon in the vicinity of the selected location. ' 
For example, the special icon is placed in the margin by 
the selected location. 

In a step 34, upon a user selecting the annotation, 

*o the associated annotation dialog box is displayed. For 
example, the annotation is selected by selecting the 
special icon. Once the annotation has been selected, 
the user may then read the annotation, modify the an- 
notation, delete the annotation, and so on. 

«s Additionally, the user may move the location of the 
annotation, for example by changing the location of the 
special icon. For example, the special icon is cut/copied 
and pasted to a new location. Alternately, the special 
icon is moved by a "drag-and-drop' technique, or by 

so some other method. 

Figures 3, 4, 5, 6, 7, 8 and 9 show various screens 
of computer display 12 which illustrate interaction with 
a document 15 in order to place and manipulate anno- 
tations in accordance with a preferred embodiment of 

55 the present invention. The computer screen shown in 
Figure 3 includes in addition to document 15, various 
interface buttons on a toolbar 23, including an annotate 
button 22. Document 15 may be, for example, an on- 
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line book, on-line documentation, an on-line manual, a 
text file, a graphics file or any document which contains 
any combination of text and/or graphics. 

A cursor 24 is under control of an input device such 
as mouse 1 4 or a trackball input device, an isopoint input 
device, a trackpad input device, cursor keys or some 
other input device either separate from or incorporated 
as part of keyboard 13. In order to annotate a location 
in document 15, cursor 24 may be used to select anno- 
tate button 22. Alternately, in order to annotate a loca- 
tion, another action may be taken, for example, by se- 
lecting an 'annotation* listing on a pull-down menu, de- 
pressing a particular function key on a keyboard, de- 
pressing a particular sequence of keys on a keyboard, 
or some other pre-defined action 

Upon selection of annotate button 24, the annota- 
tion feature is indicated. For example, Figure 4 shows 
that cursor 24 has been replaced by a special paper clip 
shaped cursor 44. As discussed above, there other 
ways selection of the annotation feature may be indicat- 
ed. 

Inthe preferred embodiment the shape of the cursor 
is defined using standard library functions, such as 
LoadCursor () and SetCursor () available within Micro- 
soft Windows operating system, available from Micro- 
soft Corporation having a business address of One Mi- 
crosoft Way, Redmond, Washington 98052. 

As shown in Figure 4, special paperclip shaped cur- 
sor 44 is used to select a location within document 15. 
For example, the location is selected by placing special 
paperclip shaped cursor 44 overthe location within doc- 
urnei : ' 1 5 and jessing a key on mouse 14. 

Upon selects if the location within document 15, 
an annotation box is ihown. For example, as shown in 
Figuro5, annotation text is contained within an annotate, 
dialog box 51. Annotate dialog box 51 includes, for ex- 
ample, an annotate text section 52. Using keyboard 13 
or some other input method, such as speech recogni- 
tion, etc., a user may input an annotation into annotate 
text section 52 Annotate dialog box 51 may be closed, 
for example, by using cursor 24 to select an "OK" button 
53. Selection of a delete button 54 deletes annotation 
dialog box 51. Selection of a cancel button 55 closes 
annotate dialog box 51 and cancels the action taken 
which resulted in annotation dialog box 51 being dis- 
played. 

In the preferred embodiment, objects are attached 
to dialog box 51 using a "drag-and-drop* operation. Ai-' 
temately, an additional 'Attach button' is added to an- 
notate dialog box 51 . Objects attached to the annotate 
text section 52 appear as an icon. Fa example, in Figure 
7, sound icon 59 appears in annotate text section 52. 
Double-clicking on sound icon 59 results in a recorded 
message being played back. The attaching of objects to 
annotate text section 52 is implemented using standard 
library functions available within the Microsoft Windows 
operating system. For example, these library functions 
include Windows Object Linking and Embedding (OLE) 



functions. 

Once 'OK' button 53 has been selected and anno- 
tate dialog box 51 has been closed, the selected location 
is marked, for example by displaying a special icon 61 

s in the vicinity of the selected location, as 6hown in Figure 
6. For example, as shown in Figure 6. special icon 61 is 
placed in the margin of document 15 near the selected 
location. Alternatively, special icon 61 may be inserted 
directly into the selected location, moving adjacent text 

» apart to fit in special icon 61 . Another alternative is to 
place special icon 61 elsewhere on computer display 1 2, 
such as at the top ol document 15. In this case special 
icon 61 is associated with the selected location, for ex- 
ample, by a label next to special icon 61 or by a pop-up 

'5 label that appears when cursor 24 Is placed over special 
icon 61. 

In the preferred embodiment, standard library func- 
tions available within the Microsoft Windows operating 
system are used both to display document 15 and spe- 

20 cial icon 61. For example, to display document 15, 
DrawText () and TextOut 0 can be used. To display spe- 
cial icon 61 , CreateWindow 0 can be used to create an 
owner drawn button and Drawlcon 0 or BitBIt 0 can be 
used to draw the icon onto the button. Alternatively, Dra- 

25 wlconOorBitB»Ocanbeusedtodrawasymboland 
then mouse button messages, such as 
WMJ.BUTTONDBCLCLK, are monitored. 

The standard library functions space can be used 
to create space as needed for special icon 61 , display 

30 special icon 61 as part of document 15 and detect 
mouse selections of special icon 61 . The same standard 
library functions are used to display ?nnotate dialog box 1 
51. 

In an alternative embodiment, special icon 61 is dis- 
ss playedbycreatingasrnallanrratat™ 
" icon 61 and displaying the special annotation window 
on top of the display for display document 1 5. The dis- 
play of special icon 61 is then synchronized using cus- 
tom functionality placed within the operating system. 
4* In another alternative embodiment, special icon 61 
is displayed using a separate annotation program that 
operates separately from the interface for document 15. 
The separate annotation program creates a special an- 
notation window for special icon 61 and displays the 
45 special annotation window on top of the display for dis- 
play document 15. Thedisplay of special icon61 is syn- 
chronized using dynamic data exchange (DDE) availa- 
ble within the Microsoft Windows operating system. 
Upon special icon 61 being selected, dialog box 51 
so reappears, as shown in Figure 7. Special con 61 may 
be selected, for example, by positioning cursor 24 over 
special icon 61 and 'double-clicking' a key on mouse 
14. Once annotation dialog box 51 is displayed, the user 
may then read the annotation, modify the annotation, 
« delete the annotation, and so on. 

Multiple annotations may beadded to document 15. 
For each annotation, a new marking is added. For ex- 
ample, Figure B shows, in addition to special icon 61, 



an additional special icon 62. The annotation represent- 
ed by special icon 62 may have been added to docu- 
ment 15, for example, by performing the steps set out 
in Figure 2 and illustrated by Figures 3 through 6. Alter- 
nately, the annotation represented by special icon may s 
have been added in another way, for example by dupli- 
cating an existing annotation. 

In the preferred embodiment, copy, cut and paste 
features are implemented for text, graphics and other 
annotations within annotate text section 52, using stand- io 
ard library functions available within the Microsoft Win- 
dows operating system. For text, SendMessage () can 
be used to send WM.COPY, WM.CUT, and 
WM_PASTE messages. For graphics, frequently used 
functions include OpenClipboard 0. CreateBitmap Q, » 
SetClipboardData 0, GetClipboard Data (), GetBitmap- 
Bits (), GetObject (), CbseClipboard (), and BHBIt Q. 

The location of each annotation may be moved. 
This is illustrated by Figure 9, which shows the location 
of the annotation represented by special icon 62 having a> 
been moved to a new location in document 15. The 
movement has taken place, for example, by 'dragging 
and dropping* special icon 62 to the new location. In a 
drag and drop operation, cursor 24 is positioned over 
special icon 62. A button on mouse 14 is then depressed & 
and held depressed while cursor 24 is positioned over 
a new location. Upon release of the button on mouse 
1 4, the annotation is moved to the new location and spe- 
cial icon 62 is displayed in close vicinity to the new lo- 
cation, for example in the margin of document 15 at a so 
position adjacent to the new location. Annotations may 
be moved by other means, for example, by cutting or 
copying the annotation and pasting the annotation to a 
new location. 

. In th'^ preferred embodiment, copy, cut, paste and 35 
drag-and-drop features are implemented, for annota- 
tions, using standard library functions available within 
the Microsoft Windows operating system. Functions for 
copy, cut and paste features are listed above. For drag- 
and-drop operations, the additional functions include *o 
DragAcceptRles 0. DragQueryFile 0, DragQueryPomt 
0, and DragFinish 0- 

Figure 10 is a flowchart which describes operation 
of an interface for annotating document 15 in accord- 
ance with an alternate preferred embodiment of the 45 
present invention. In a step 91 , upon selection by a user 
of a location in document 15 arid the annotation feature, 
an annotation box is displayed. For example, the kxa-~" 
tion is selected by placing the cursor over the location 
and depressing or releasing a key on an input device so 
such as a mouse, trackball, trackpad or keyboard. The 
annotation feature Is selected, for example, by selecting 
a special 'annotation" button. Alternately, the selection 
of the annotation may be made, for example, by select- 
ing an "annotation' listing on a pull-down menu, by de- 55 
pressing a particular function key on a keyboard, by de- 
pressing a particular sequence of keys on a keyboard, 
or by some other means. The text for the annotation is 
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displayed, for example, within an annotation dialog box. 
Initially, a displayed annotation box does not have any 
usewtefined text The user may add the desired text for 
the annotation, for example using a keyboard or another 
input device. 

In a step 92. upon closing the annotation box, the 
selected location is marked, for example by displaying 
a special icon in the vicinity of the selected location. For 
example, the special icon is placed in the margin of doc- 
ument 15 near the selected location. Upon a user se- 
lecting the annotation , the associated annotation dialog 
box is displayed. For example, the annotation is select- 
ed by selecting the special icon. Once the annotation 
has been selected, the user may then read the annota- 
tion, modify the annotation, delete the annotation, and 
so on. Additionally, the user may move the location of 
the annotation, as described above. 

Figures 11 and 1 2 show various screens of compu- 
ter display 12 which illustrate interaction with document 
15 in order to place annotations in accordance with the 
alternate preferred embodiment of the present inven- 
tion. The computer screen shown in Figure 10 includes 
document 15, various interface buttons on tool bar 23 
including annotate button 22, as described above. 

As shown in Figure 11, cursor 24 is used to select 
a location within document 1 5. For example, the location 
is selected by placing cursor 24 over the location within 
document 15 and depressing a key on mouse 14. A 
marker 104, is placed in document 15 as shown. After 
selecting the location, cursor 24 is used to select anno- 
tate button 24. Alternately, another action may be taken 
to select the annotation feature. For example;^ user 
may select an "annotation" listing on a pull-down menu, 
depressing a particular function key on a keyboard, de- 
pressing a particular sequent** of Kays on a keyboard, 
or some other pre-defined action. 

Upon selection of the annotation feature within doc- 
ument 1 5, an annotation box is shown. For example, text 
for the annotation is contained within an annotate dialog 
box. as discussed above. Once a user has input the de- 
sired text for the annotation, annotate dialog box is 
closed. The selected location is then marked, for exam- 
ple by displaying a special icon 11 1 in the vicinity of the 
selected location, as shown in Figure 12. For example, 
as shown in Figure 12. the special icon is placed in the 
margin by the selected location. 

In the preferred embodiment, standard library func- 
tions available within the Microsoft WIiWowb operating" 
system are used to display marker 104 and later special 
icon 111. For example, these functions include Draw- 
Icon 0 and BitBIt (). Position of the marker is maintained, 
for example, by the application program controlling doc- 
ument 15. 

The foregoing discussion discloses and describes 
merely exemplary methods and embodiments of the 
present invention. As will be understood by those famil- 
iar with the art, the invention may be embodied in other 
specific forms without departing from the spirit or essen- 
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tial characteristics thereof. Accordingly, the disclosure 
of the present invention is intended to be illustrative, but 
not limiting, of the scope of the invention, which is set 
forth in the following claims. 



Claims 

1 . A computer implemented method for annotating a 
document (1 5) comprising the steps of: io 

(a) upon selection of an annotation feature and 
a location in the document (15), displaying an 
annotation box (51); and, 

(b) upon closing the annotation box (51 ), mark- is 
ing the location in the document (15). 

2. A computer implemented method as in claim 1 
wherein step (a) includes the following substeps: 

so 

(al) upon selection of the annotation feature, 
indicating the annotation feature; and, 
(a.2) upon selecting the location in the docu- 
ment (15), displaying the annotation box (51). 

25 

3. A computer implemented method as in claim 2 
wherein substep (a.1) includes changing a shape 
of a cursor (24,44). 

4. A computer implemented method as in claim 1 a> 
wherein in step (b) the location is marked by dis- 
playing a special icon (61,62) in a location in a vi- 
cinity of the location in the document (15). 

5. Acc^uterimplemi^ied methcxiasfc.daim4ad- as 
ditionally comprising u ie following step: 

(c) upon selection of the special icon (61,62), 
displaying the annotation box (51). 

6. A computer implemented method as in daim 4 ad- <o 
ditionally comprising the following step: 

(c) in response to a user, moving the special 
icon (61 ,62) to a new location within the document 
(15). 

45 

7. A computer implemented method as in claim 1 
wherein step (a) includes the following substeps: 

(a.1) upon selecting the location in the docu- 
ment (1 5), showing a marker (104) in the doc- so 
ument (1 5) at the selected location; and, 
(a.2) upon selection of the annotation feature, 
displaying the annotation box (51). 

8. A computer implemented as in claim 1 additionally ss 
comprising the following step performed after step 

(a): 

in response to a user selection, attaching 



graphic information to the annotation box (51 ). 

9. A computer implemented as in daim 1 additionally 
comprising the following step performed after step 
(a): 

in response to a user selection, attaching a 
sound file to the annotation box (51 ). 

10. A computer implemented as in claim 1 additionally 
comprising the following step performed after step 
(a): 

in response to a user selection, attaching a 
movie file to the annotation box (51). 
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UPON SELECTION OF ANNOTATE 
FEATURE, INDICATE THE ANNOTATE 
FEATURE 



UPON SELECTING A LOCATION IN THE 
DOCUMENT, DISPLAY ANNOTATION 



UPON CLOSING THE ANNOTATION, 
MARKING THE SELECTED LOCATION !N 
THE DOCUMENT 



UPON SELECTING THE ANNOTATION 
LOCATION, DISPLAYING THE 
ASSOCIATED ANNOTATION 
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